feat(visualizer): device-aware shell rendering with Remotion Player#2023
feat(visualizer): device-aware shell rendering with Remotion Player#2023
Conversation
There was a problem hiding this comment.
Pull request overview
This PR migrates the visualizer playback pipeline from a Pixi.js canvas renderer to a Remotion-based player, adding device-aware shells and branded cyberpunk visual effects for both preview and video export. It also plumbs deviceType through execution dumps into the visualizer so the correct shell can be selected.
Changes:
- Replaced the Pixi.js visualizer
Playerwith@remotion/playerand introduced Remotion compositions/scenes (opening, steps, ending, progress bar). - Added a shared frame-timeline model (
ScriptFrame/FrameMap) and shared state derivation for Remotion preview + Canvas-based export. - Plumbed
deviceTypefrom core dumps → report store → visualizer player, and added a persistedeffectsEnabledpreference.
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Adds Remotion dependencies and removes Pixi-related packages from the lockfile. |
| packages/visualizer/src/utils/replay-scripts.ts | Extends replay metadata to include deviceType. |
| packages/visualizer/src/utils/pixi-loader.ts | Removes the Pixi texture loader (Pixi renderer is removed). |
| packages/visualizer/src/store/store.tsx | Adds persisted effectsEnabled preference. |
| packages/visualizer/src/hooks/usePlaygroundExecution.ts | Refactors hook signature to an options object and plumbs deviceType into grouped dumps. |
| packages/visualizer/src/component/universal-playground/index.tsx | Updates usePlaygroundExecution callsite to pass options object + deviceType. |
| packages/visualizer/src/component/playground-result/index.tsx | Passes deviceType through to the visualizer Player. |
| packages/visualizer/src/component/player/remotion/visual-effects.ts | Adds pure computation utilities for effects + device shell resolution/layout constants. |
| packages/visualizer/src/component/player/remotion/frame-calculator.ts | Introduces FrameMap/ScriptFrame timeline calculation for Remotion + export. |
| packages/visualizer/src/component/player/remotion/export-branded-video.ts | Adds Canvas+MediaRecorder-based branded replay export with shells and effects. |
| packages/visualizer/src/component/player/remotion/derive-frame-state.ts | Shared accumulator-based frame state derivation used by Remotion preview + export. |
| packages/visualizer/src/component/player/remotion/StepScene.tsx | Implements the Remotion “steps” scene with shells/effects/cursor/ripple/overlays. |
| packages/visualizer/src/component/player/remotion/ProgressBar.tsx | Adds a Remotion progress bar overlay. |
| packages/visualizer/src/component/player/remotion/OpeningScene.tsx | Adds cyberpunk opening scene. |
| packages/visualizer/src/component/player/remotion/EndingScene.tsx | Adds cyberpunk ending scene. |
| packages/visualizer/src/component/player/remotion/CyberOverlays.tsx | Adds shared overlay components/icons used across scenes. |
| packages/visualizer/src/component/player/remotion/BrandedComposition.tsx | Wires opening/steps/ending/progress into a single Remotion composition. |
| packages/visualizer/src/component/player/index.tsx | Replaces Pixi player with Remotion player; adds export, effects toggle, speed control integration. |
| packages/visualizer/src/component/player/index.less | Updates player styling for Remotion output + custom controls. |
| packages/visualizer/src/component/blackboard/index.tsx | Replaces Pixi-based blackboard overlays with DOM/CSS overlays and click coordinate mapping. |
| packages/visualizer/src/component/blackboard/index.less | Adds CSS overlay styling + pulse animation for blackboard highlights/points. |
| packages/visualizer/package.json | Removes Pixi deps, adds remotion and @remotion/player. |
| packages/core/src/types.ts | Adds optional deviceType to IGroupedActionDump + serialization. |
| packages/core/src/agent/agent.ts | Populates deviceType on grouped dumps from interfaceType. |
| apps/report/src/components/store/index.tsx | Stores/propagates deviceType from scripts info into report UI state. |
| apps/report/src/components/detail-panel/index.tsx | Passes deviceType into the visualizer Player. |
| apps/report/src/App.tsx | Passes deviceType into the visualizer Player for the main report view. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
packages/visualizer/src/component/player/remotion/export-branded-video.ts
Outdated
Show resolved
Hide resolved
packages/visualizer/src/component/player/remotion/CyberOverlays.tsx
Outdated
Show resolved
Hide resolved
packages/visualizer/src/component/player/remotion/frame-calculator.ts
Outdated
Show resolved
Hide resolved
packages/visualizer/src/component/player/remotion/export-branded-video.ts
Show resolved
Hide resolved
packages/visualizer/src/component/player/remotion/visual-effects.ts
Outdated
Show resolved
Hide resolved
packages/visualizer/src/component/player/remotion/ProgressBar.tsx
Outdated
Show resolved
Hide resolved
Deploying midscene with
|
| Latest commit: |
8977df1
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://245e2ca7.midscene.pages.dev |
| Branch Preview URL: | https://feat-device-shell-rendering.midscene.pages.dev |
eae82e7 to
88d2c2c
Compare
…alculateFrameMap function
…ling in video export
…functionality chore(deps): remove unnecessary pixi dependencies from package.json and pnpm-lock.yaml
Use renderCustomControls API to place download, export, and settings buttons directly in the player's built-in control bar instead of a separate toolbar section below the video.
…ng in drawChromeTitleBar function
…rtrait mode support
…hance layout calculations
- Remove all cyberpunk visual effects (opening/ending scenes, glitch, ripple, cursor trail, scanlines, HUD corners, 3D transforms) - Delete OpeningScene, EndingScene, ProgressBar, CyberOverlays components - Simplify frame-calculator, StepScene, export-branded-video - Replace effectsEnabled store state with subtitleEnabled toggle - Add subtitle visibility toggle to player settings panel - Unify control bar button styles (play/pause, fullscreen, settings) - Improve sidebar icon-button hover effect to match theme toggle
Add min-width constraints and proper overflow handling to prevent page-side and main-right panels from overlapping at narrow widths.
53fb434 to
d0062c0
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 22 changed files in this pull request and generated 7 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const mainLayoutChangedRef = useRef(false); | ||
| const [sidebarWidth, setSidebarWidth] = useState(() => { | ||
| const saved = localStorage.getItem(SIDEBAR_WIDTH_KEY); | ||
| return saved ? Number(saved) : DEFAULT_SIDEBAR_WIDTH; |
There was a problem hiding this comment.
sidebarWidth is initialized from localStorage using Number(saved) without validation. If the stored value is missing/invalid, this can become NaN and result in an invalid style.width. Consider validating/clamping the parsed value (and falling back to DEFAULT_SIDEBAR_WIDTH when Number.isFinite is false).
| return saved ? Number(saved) : DEFAULT_SIDEBAR_WIDTH; | |
| const parsed = saved !== null ? Number(saved) : NaN; | |
| // Fallback to default if the stored value is missing, invalid, or non-positive. | |
| return Number.isFinite(parsed) && parsed > 0 | |
| ? parsed | |
| : DEFAULT_SIDEBAR_WIDTH; |
| a.href = url; | ||
| a.download = 'midscene_report.html'; | ||
| a.click(); | ||
| URL.revokeObjectURL(url); |
There was a problem hiding this comment.
URL.revokeObjectURL(url) is called immediately after a.click(). In some browsers (notably Safari), revoking the object URL synchronously can cancel the download before it starts. Consider revoking asynchronously (e.g., via setTimeout) or after the navigation/download has been initiated.
| URL.revokeObjectURL(url); | |
| setTimeout(() => { | |
| URL.revokeObjectURL(url); | |
| }, 0); |
packages/visualizer/src/component/player/remotion/export-branded-video.ts
Show resolved
Hide resolved
| a.click(); | ||
| URL.revokeObjectURL(url); |
There was a problem hiding this comment.
After recording stops, the captureStream() tracks are never stopped, and the object URL is revoked immediately after a.click(). Stopping the stream tracks helps avoid leaking resources, and delaying URL.revokeObjectURL avoids flaky downloads in some browsers.
| a.click(); | |
| URL.revokeObjectURL(url); | |
| // Append to DOM to improve cross-browser reliability of the click-triggered download | |
| document.body.appendChild(a); | |
| a.click(); | |
| document.body.removeChild(a); | |
| // Stop all tracks from the capture stream to avoid leaking resources | |
| stream.getTracks().forEach((track) => track.stop()); | |
| // Delay revoking the object URL to avoid flaky downloads in some browsers | |
| setTimeout(() => { | |
| URL.revokeObjectURL(url); | |
| }, 1000); |
…n up player - Rewrite Timeline component from pixi.js to Canvas 2D API - Remove pixi.js and pixi-filters dependencies - Delete pixi-loader utility module - Hide resize handle visual indicator completely - Simplify StepScene layout calculations for portrait mode - Add Remotion license acknowledgment to suppress console warning
…; remove unused components and dependencies
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 25 changed files in this pull request and generated 10 comments.
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| <span className="time-display"> | ||
| {formatTime(player.currentFrame, frameMap.fps)} /{' '} | ||
| {formatTime(totalFrames, frameMap.fps)} |
There was a problem hiding this comment.
The total time display uses totalFrames, but the player’s last reachable frame index is totalFrames - 1 (your seek percent calculation also uses totalFrames - 1). This causes the displayed duration to overshoot by ~1 frame (often showing 00:01 when the last frame is still 00:00). Consider formatting the end time using totalFrames - 1 (or compute duration seconds explicitly) to align UI with the actual timeline range.
| {formatTime(totalFrames, frameMap.fps)} | |
| {formatTime(totalFrames - 1, frameMap.fps)} |
| while (accumulated >= frameDuration) { | ||
| accumulated -= frameDuration; | ||
| const next = frameRef.current + 1; | ||
| if (next >= durationRef.current) { | ||
| if (loopRef.current) { | ||
| frameRef.current = 0; | ||
| setCurrentFrame(0); | ||
| } else { | ||
| frameRef.current = durationRef.current - 1; | ||
| setCurrentFrame(durationRef.current - 1); | ||
| setPlaying(false); | ||
| return; | ||
| } | ||
| } else { | ||
| frameRef.current = next; | ||
| setCurrentFrame(next); | ||
| } |
There was a problem hiding this comment.
This while loop can call setCurrentFrame many times in a single RAF tick (e.g., when the tab was backgrounded or playbackRate is high), causing unnecessary React renders. Consider computing the final frame advance in local variables (and whether playback should stop/loop), then calling setCurrentFrame at most once per tick.
| while (accumulated >= frameDuration) { | |
| accumulated -= frameDuration; | |
| const next = frameRef.current + 1; | |
| if (next >= durationRef.current) { | |
| if (loopRef.current) { | |
| frameRef.current = 0; | |
| setCurrentFrame(0); | |
| } else { | |
| frameRef.current = durationRef.current - 1; | |
| setCurrentFrame(durationRef.current - 1); | |
| setPlaying(false); | |
| return; | |
| } | |
| } else { | |
| frameRef.current = next; | |
| setCurrentFrame(next); | |
| } | |
| if (accumulated >= frameDuration) { | |
| const framesToAdvance = Math.floor(accumulated / frameDuration); | |
| accumulated = accumulated % frameDuration; | |
| let newFrame = frameRef.current; | |
| let shouldStop = false; | |
| for (let i = 0; i < framesToAdvance; i++) { | |
| const next = newFrame + 1; | |
| if (next >= durationRef.current) { | |
| if (loopRef.current) { | |
| newFrame = 0; | |
| } else { | |
| newFrame = durationRef.current - 1; | |
| shouldStop = true; | |
| break; | |
| } | |
| } else { | |
| newFrame = next; | |
| } | |
| } | |
| if (newFrame !== frameRef.current) { | |
| frameRef.current = newFrame; | |
| setCurrentFrame(newFrame); | |
| } | |
| if (shouldStop) { | |
| setPlaying(false); | |
| return; | |
| } |
| function deriveTaskId( | ||
| scriptFrames: ScriptFrame[], | ||
| stepsFrame: number, | ||
| ): string | null { | ||
| let taskId: string | null = null; | ||
| for (const sf of scriptFrames) { | ||
| if (sf.durationInFrames === 0) { | ||
| if (sf.startFrame <= stepsFrame) { | ||
| taskId = sf.taskId ?? taskId; | ||
| } | ||
| }; | ||
|
|
||
| // Add error handler for MediaRecorder | ||
| mediaRecorder.onerror = (event) => { | ||
| console.error('MediaRecorder error:', event); | ||
| message.error('Video recording failed. Please try again.'); | ||
| this.recording = false; | ||
| this.mediaRecorder = null; | ||
| }; | ||
|
|
||
| this.mediaRecorder = mediaRecorder; | ||
| this.recording = true; | ||
| return this.mediaRecorder.start(); | ||
| } | ||
|
|
||
| stop() { | ||
| if (!this.recording || !this.mediaRecorder) { | ||
| console.warn('not recording'); | ||
| return; | ||
| continue; | ||
| } | ||
|
|
||
| // Bind onstop handler BEFORE calling stop() to ensure it's attached in time | ||
| this.mediaRecorder.onstop = () => { | ||
| // Check if we have any data | ||
| if (this.chunks.length === 0) { | ||
| console.error('No video data captured'); | ||
| message.error('Video export failed: No data captured.'); | ||
| return; | ||
| } | ||
|
|
||
| const blob = new Blob(this.chunks, { type: 'video/webm' }); | ||
|
|
||
| // Check blob size | ||
| if (blob.size === 0) { | ||
| console.error('Video blob is empty'); | ||
| message.error('Video export failed: Empty file.'); | ||
| return; | ||
| } | ||
|
|
||
| const url = URL.createObjectURL(blob); | ||
| const a = document.createElement('a'); | ||
| a.href = url; | ||
| a.download = 'midscene_replay.webm'; | ||
| a.click(); | ||
| URL.revokeObjectURL(url); | ||
| }; | ||
|
|
||
| this.mediaRecorder.stop(); | ||
| this.recording = false; | ||
| this.mediaRecorder = null; | ||
| if (stepsFrame < sf.startFrame) break; | ||
| taskId = sf.taskId ?? taskId; | ||
| } | ||
| return taskId; | ||
| } |
There was a problem hiding this comment.
deriveTaskId linearly scans scriptFrames. Since it’s invoked from an effect that runs every frame update, this becomes O(N) work per frame and can get expensive for larger replays. Consider precomputing a frame→taskId lookup (e.g., an array sized totalDurationInFrames or a list of segment boundaries with binary search), or track the current script index incrementally as playback progresses.
| const [controlsVisible, setControlsVisible] = useState(true); | ||
| const hideTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null); | ||
|
|
||
| const fadeOutItem = async ( | ||
| graphics: PIXI.Container | PIXI.Graphics | PIXI.Text, | ||
| duration: number, | ||
| frame: FrameFn, | ||
| ): Promise<void> => { | ||
| return fadeInGraphics(graphics, duration, frame, 0); | ||
| }; | ||
| const showControls = useCallback(() => { | ||
| setControlsVisible(true); | ||
| if (hideTimerRef.current) clearTimeout(hideTimerRef.current); | ||
| hideTimerRef.current = setTimeout(() => setControlsVisible(false), 3000); | ||
| }, []); |
There was a problem hiding this comment.
The auto-hide timer isn’t cleared on unmount. If the component unmounts while a timeout is pending, it may attempt to call setControlsVisible after unmount. Add an effect cleanup that clears hideTimerRef.current (and consider reusing the same cleanup for the mouse-leave timer).
| const canvas = document.createElement('canvas'); | ||
| canvas.width = W; | ||
| canvas.height = H; |
There was a problem hiding this comment.
Export hard-codes a 16:9 canvas (960x540). This will distort or crop portrait replays (e.g., iPhone/Android), which is a key scenario in this PR. Consider deriving export dimensions from frameMap.imageWidth/imageHeight (or from a device-aware composition size) and updating the camera math accordingly, so exported video matches the same aspect ratio/device shell as the preview.
| const hasPtrData = | ||
| Math.abs(camera.pointerLeft - Math.round(baseW / 2)) > 1 || | ||
| Math.abs(camera.pointerTop - Math.round(baseH / 2)) > 1 || | ||
| Math.abs(prevCamera.pointerLeft - Math.round(baseW / 2)) > 1 || | ||
| Math.abs(prevCamera.pointerTop - Math.round(baseH / 2)) > 1; |
There was a problem hiding this comment.
hasPtrData compares pointer positions against baseW/baseH centers, but pointer coordinates are derived from script/image-specific dimensions (imgW/imgH and camera). When scripts contain mixed image sizes (or when base dimensions differ), this can incorrectly hide/show the cursor. Consider comparing to the effective image center for the current frame (e.g., Math.round(imgW/2) / Math.round(imgH/2)) or a dedicated 'no pointer data' sentinel.
| const hasPtrData = | |
| Math.abs(camera.pointerLeft - Math.round(baseW / 2)) > 1 || | |
| Math.abs(camera.pointerTop - Math.round(baseH / 2)) > 1 || | |
| Math.abs(prevCamera.pointerLeft - Math.round(baseW / 2)) > 1 || | |
| Math.abs(prevCamera.pointerTop - Math.round(baseH / 2)) > 1; | |
| const centerX = Math.round(imgW / 2); | |
| const centerY = Math.round(imgH / 2); | |
| const hasPtrData = | |
| Math.abs(camera.pointerLeft - centerX) > 1 || | |
| Math.abs(camera.pointerTop - centerY) > 1 || | |
| Math.abs(prevCamera.pointerLeft - centerX) > 1 || | |
| Math.abs(prevCamera.pointerTop - centerY) > 1; |
| const stream = canvas.captureStream(fps); | ||
| const recorder = new MediaRecorder(stream, { mimeType: 'video/webm' }); |
There was a problem hiding this comment.
Using a fixed mimeType: 'video/webm' can throw at construction time if unsupported (and some browsers don’t support WebM/MediaRecorder at all). Consider guarding with MediaRecorder.isTypeSupported(...) and falling back to a supported type (or surfacing a clear message). Also, revoking the object URL immediately after a.click() can cancel downloads in some browsers; consider deferring URL.revokeObjectURL(url) (e.g., via setTimeout) or revoking after a user gesture completes.
| allScreenshots.map(async (shot, index) => { | ||
| if (imgCache.has(shot.img)) { | ||
| // Compute layout from cached image | ||
| const img = imgCache.get(shot.img)!; | ||
| const w = Math.floor( | ||
| (screenshotMaxHeight / img.naturalHeight) * img.naturalWidth, | ||
| ); | ||
| allScreenshots[index].x = leftForTimeOffset(shot.timeOffset); | ||
| allScreenshots[index].y = screenshotTop; | ||
| allScreenshots[index].width = w; | ||
| allScreenshots[index].height = screenshotMaxHeight; | ||
| return; |
There was a problem hiding this comment.
This mutates allScreenshots items (which originate from props.screenshots). Mutating props can cause hard-to-track rendering inconsistencies and breaks React’s immutability expectations. Consider keeping a separate internal layout array/map (e.g., in stateRef.current) keyed by screenshot id, or copying props.screenshots into a new array before assigning layout fields.
| <img | ||
| src={img} | ||
| style={{ | ||
| width: w, | ||
| height: h, | ||
| transformOrigin: '0 0', | ||
| transform: transformStyle, | ||
| }} | ||
| /> |
There was a problem hiding this comment.
Decorative <img> elements used purely for visual playback should include alt=\"\" (empty alt) so screen readers don’t announce the raw URL / redundant content. Apply the same to other cursor/spinner image elements in this component.
| const handleClick = (e: React.MouseEvent<HTMLDivElement>) => { | ||
| if (!props.onCanvasClick || !containerRef.current) return; | ||
| const rect = containerRef.current.getBoundingClientRect(); | ||
| const scaleX = screenWidth / rect.width; | ||
| const scaleY = screenHeight / rect.height; | ||
| const x = Math.round((e.clientX - rect.left) * scaleX); | ||
| const y = Math.round((e.clientY - rect.top) * scaleY); | ||
| props.onCanvasClick([x, y]); | ||
| }; |
There was a problem hiding this comment.
Click→image coordinate mapping uses the container’s bounding box, but the screenshot <img> has its own border and the container may include overlay/layout differences. This can introduce small but consistent coordinate offsets. Consider measuring the actual screenshot image element bounds (e.g., attach a ref to .blackboard-screenshot) and compute coordinates relative to that rect for more accurate hit positions.
Summary
deviceTypeproprenderCustomControlsAPIeffectsEnabledtoggle and playback speed settingsTest plan